Applying meta-analytical procedures to software engineering experiments

Deriving reliable empirical results from a single experiment is an unlikely event. Hence to progress multiple experiments must be undertaken per hypothesis and the subsequent results e ectively combined to produce a single reliable conclusion. Since results are quantitative in nature, a quantitative conclusion would be the optimal solution. Other disciplines use meta-analytic techniques to achieve this result. The treatise of this paper is: can metaanalysis be successfully applied to current Software Engineering experiments? The question is investigated by examining a series of experiments, which themselves investigate | which defect detection technique is best? Applying metaanalysis techniques to the Software Engineering data is relatively straightforward, but unfortunately the results are highly unstable, as the meta-analysis shows that the results are highly disparate and don't lead to a single reliable conclusion. The reason for this de ciency is the excessive variation within various components of the experiments. The paper outlines various ideas from other disciplines for controlling this variation and describes a number of recommendations for controlling and reporting empirical work to advance the discipline towards a position where meta-analysis can be pro tably employed.